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ABSTRACT 

While the expansion of worldwide hypermedia systems 
has opened new gateways in the information industry, the result has 
been material being archived faster than it can be read. Users need 
to be able to scan information quickly and have more control in 
information seeking and retrieval. A new area of research is in the 
development of multimedia tools that answer these needs. One of the 
aims in the development of MUSLI (MUlti Sensory Language Interface) 
is to integrate dynamic abstract symbolism into hypermedia technology 
(linked multimedia documents embedded in a network), allowing for 
more efficient information retrieval and communication. In a MUSLI 
movie, users can stop and display additional information as they need 
an! skim condensed versions of a document, searching for relevant 
sections before breaking down the symbols to display pertinent 
information. MUSLI documents incorporate material processes 
("doing"), mental processes ("sensing"), and relational processes 
("being"); these processes are brought together with dynamic symbols. 
The proposed system will be flexible enough to blend traditional 
forms of documentation into modern multimedia information retrieval 
systems while gradually becoming enriched with its own symbolic 
language. (Contains 22 references.) (AEF) 
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Abstract: While the expansion of worldwide hypermedia systems has opened new 
gateways to an undreamt-of commerce in information, this has led to valuable 
maierial bemg archived faster than it can possibly be read. In this paper we suggest 
as a new and vitally important area of research, the development of multimedia tools 
that allow more efficient and effective "reading" and "writing" using new types of 
hypermedia documents. We look beyond traditional hypertext, taking the first steps 
towards a language of dynamic abstract symbols that will help authors to express 
precisely and concisely both concrete and abstract ideas. MUSLI readers will enjoy 
searching for detailed information by invoking new multimedia "speed reading- 
techniques. 



1. Introduction 



In spite of, and partly because of, dramatic advances in information technology, keeping up to date in anv 
d. cphne IS now almost an impossibility. We know that hypertext systems are fv ilabfe S offe^ertain 

^^^^z^:::^' "^^rr ^'^^"^^^^ ^'p^^- scherbakov!7s;Livi;T 

iNielsen 1990). However we need more help than systems such as these are currently providing 

Although Socrates in Plato's Phaedrus argued that writing a language destroys it, readingText is certainly a 
faster method for gaining information than listening to an orator! Yet we now have a gen a ion S voun 
people many of whom find reading "too slow". People need to be able to scan information af^ s good soeeS 
readers can scan printed text, or even better. We need to have more control over our c mm nTc ing media' 
1 of information we feel motivated to reach, and the pace we wish to set ourselves. Bo^a leisure nd a 

impo nance of visual information must not be underestimated. In particular, certain types of anim S pi" ,u re 
may v 11 be worth more than a large number of static ones. Unfortunately we are stnLeverely Hm ted bTthe 
lack of tools for producing documents that involve all the senses. One of the aims o MnsTi rMni , t ^ 
Language interface) is to integrate dynamic abstract symbolism ("ablt^acfmovie Tin CrmeC hn^^^^^^^^^ 
allowing the efficient expression of ideas difficult to communicate otherwise '^VP'^^'^dia technology. 

We restnV, ^T! 7 'T :hypermedia" to mean linked multimedia documents embedded in a network 

We restrict ourselves to exploring asynchronous communication in hypermedia systems even thTugh 

-^^'-1 ^hoice w. .p, .since U is 

r '--8-8'= '"'"f-<=.i-PPropria.c,n another sense -nuUitious cuisine rrmLa^^^^^ ^ ^^UlU-Sen.sory 
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synchronous communication may well give rise to still more exciting and challenging r^^search. For example, 
Maurer and Carlson (1992) point out that although our second most important sensory organ, the ear, has a 
counterpart, the mouth, our eyes have no such counterpart. We cannot project menial images for other people to 
capture. The authors suggest that hypermedia systems may develop to a stage where we can produce concrete 
and abstract projections of our mental images, by computer, so easily and naturally that they will provide us with 
a new dimension in communication capable of transforming our lives. As wide-area and broad-band networks 
expand we have an increasingly powerful medium for communication and collaboration, whose impact will not 
be simply to tie information together but, much more significantly, to "tie people together" (Maurer, 1993a). 
This, coupled with the fact that multimedia machines are becoming increasingly powerful and affordable, 
implies that we have the mechanics for truly interactive hypermedia communication systems. Tools for 
generating new types of hypermedia documents must be developed so that we can make more efficient use of 
information than is currently possible. 

In this paper we shall introduce two new ways of "writing" multimedia material in forms that will aid 
"reading". We shall expand the idea of browsing and lake the first steps toward introducing a more flexible 
system for conveying information - MUSU. 

Humanity has evolved so many ingenious systems of communication since civilisation first dawned thai it 
makes sense to look briefly at a few of these before we propose a new one. 

2. Alternative Forms of Communication 

One of the most striking features about communication using signs and symbols is the tremendous variety 
in medium and form - from the simplest runes on clay or parchment to the most intricate designs on human skin. 
Prehistorians believe that many ancient cave drawings found in France can be considered as forms of writing: 
they have discovered groups of drawings which taken together form codes (Jean, 1989). Paintings in the 
Renaissance were richly redolent with emotionally charged symbolism. Carvings on wood, stone and metal 
range from concrete signs to abstract symbolism. Signalling devices range from smoke signals to systems of 
bells, mirrors and musical instruments, to semaphore and electric morse-code - and in the future we are bound to 
discover new options using digitised sound. Cistercian monks who practise vows of silence have developed 
highly refined systems of gestures (Umiker-Sebeok, & Sebeok, 1987). Many signs used by deaf people today 
(Jeanes, 1982) are based on these and it is significant that in many respects signing can convey meaning more 
quickly than spoken language. 

3. The Basic Idea 

MUSLI combines hypermedia techniques with the novel idea of dynamic abstract symbols. Wc aim to 
facilitate communication by developing a system in which ideas can be expressed as clearly, precisely, and 
artistically as desired while giving readers the deep satisfaction of using a rich and highly meaningful medium 
ihey can control. This paper is to be seen as the very first step, an initial coniribuiion ihai we hope will 
encourage scientists to work at opening up new realms in presenting, archiving and communicating ideas. 

Obviously any new human endeavour that is worthwhile requires an initial investment of effort. learning 
any new language demands dedication and since the system we propose includes abstract symbols it is certain to 
be no exception. 

It is a well known fact that good readers are often disappointed when they see the film version of a book 
they have read. The problem is often due to the fact that while books leave room for personal inlerpretaiion and 
imagination, movies do not. The "beautiful mountain" of the book will "look different" in the minds of diffcrenl 
readers, bul all viewers of a movie see, inevitably, the same "beaiuiful" mountain. What has happened is that 
there has been a shift in the level of abstraction - the abstract notion "beautiful mountain" has been replaced by a 
concrete version of a mountain. In MUSLI the proportion and level of abstraction can be chosen lo fit ihe 
situation, varying from abstract to very concrete as the movie producer feels is appropriate to communicate 
exactly what is desired. 

It is an interesting observation that ideas can be thought of as having the form of a mountain (Maurer. 
1992) and it is significant that we talk about having a "point of view" on a topic. As an example of looking at 
something in more than one way consider the well known problem in first year computer science courses of 
calculating the Fibonacci .scries. Many books use this example to teach recursion and then almost apologise for 
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memion.ng thai ii can be solved betler using iicrationl. Very few lecturers indeed show a third way „f looking 
at the problem using dynamic programming. Yet each of the three methods provides insights that arc valid and 
complementary views of the problem. In an interactive multimedia .system students can compare ideas lo an 
even greater extent than ever before. 

In any multimedia presentation readers should certainly be able to stop and start at will, rewind review 
and then resume. The system should al.so enable readers to master new material by building up additional skills 
as tdey go. In a MUSLI movie (i.e., a hypermedia document that includes special moving abstract symbols) 
users can stop and display additional information - at any time they feel ready for it. As explained in Section 7 
any group of symbols can be condensed to a single "short-form", and double-clicking on the composite symbol 
will decompose it again. This gives the user significant control over the amount of data on the screen at any 
particular time. The user will be able to obtain additional detailed information about scenes, characters ideas 
T:Zl'^l ''I'J'^'y '° "1'="!;.^' ^"'^''y ^'^^^"hing their memory if they have forgotten the meaning of 

book? ' ' '"''^ ^^^^ documents significant advantages over traditional 

'^"'■^ significantly, users will be able to skim condensed versions of a document, .searching for relevant 
sections before decomposing the symbols to di.splay pertinent information. With facilities such as these it will 
be possible lo speed-read MUSLI documents faster than printed text. 

As .sy.stems become available that include electronic personal assistants (Cypher, 1993- Maurer 1993b- 
Z..SSOS. & Witten. 1985) readers of MUSLI documents will obtain further help\n over'comVngThe wides re 'd 
problem of information overload. The system will "learn" what level of detail needs to be displayed so that the 
reader can access needed information more efficiently. 

4. What MUSLIis Not 

The system we propose must not be misunderstood. It enhances traditional multimedia documents We 
propose to extend communication, not restrict it! MUSLI is not simply a system of cartoons - cartoons are 
designed 10 be understood without any investment in learning and are quite close to a simplified reality MUSLI 
is not a silent movie or some system of signs. It will make appropriate use of computer graphics (2D and 3D) 
cartoons, audio clips (music and speech Doth analogue and digital), video clips (analogue and digital ,' 
photographs (digitised), animation including 3D modelling; and of course text. Naturally there will still be 
movies - we delight in the work on human animation being done by scientists and artists such as those at the 
MIRALab at the University of Geneva (Magnenat-TTialmann, & Thalmann, 1991). Visualisation will continue 
o be an important means of communication (IEEE, 1993) and we watch with interest the work being done in the 
grandlelign^MSu ^' °^ achievements will be eventually integrated into a single 

5. Using Sound, Speech and Text 

Sound can frequently be used to good effect. There is hardly a better way of setting moods for scenes and 
m^^h A t or^f ^""y thunderstorm than by using appropriate sounds. An interesting point 
mad by Al y (1993 , is that peripheral or parallel streams of information containing redundant inform a. fon can 
be ol importance in helping students understand complex ideas. 

Although speech is inherently slow, there may be alternative ways of processing it. We note, for example 

rnllhi T P°f' ^'V°'.T^"'"' '° ^'"^ 'P'"'^ '"'^"^"^ ^P''-'=d still maintain an 

acceptable level of pilch and clarity. 

One interesting idea, that should be considered further, is to extend the use made of digitised speech For 
example for scanning purposes, users could display on the screen the waveform patterns of the speech It mas- 
be possible to search for particular sections to play by learning to "read" waveforms, at least in a broad sense- for 
example, changes in volume may indicate passages of significance. If the wave patterns are expanded and' the 

^ Some warn of arithmetic overflows though what really overflows is lime not ^t/cI If m - rrn\ ,i '.i .-> 

n„b„. ,1,.. ,h. f„.o,i.„ ,.„c,i„„ r,„: i,„g„> i„„.„r(;i) r "uJil t SJ;; 'X' 
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ampliiudc is displayed in, say, red while ihe pilch is displayed in blue, additional information can be gained. In 
particular, it should be possible to search digitised speech more efficiently than at present. For example, 
alternating passages of high and low pitch may well mark dialogue. Exciting passages will stand out. Sentences 
will be seen marked off from one another and other information will be visible such as the occurrence of a 
question. If the author of the document provides signposts in the form of icons alongside key passages the users 
will be further aided in their searches. It should also be possible for specialised computer programs to scan the 
material in greater detail. With methods such as these authors will be able to produce documents allowing 
"readers" to understand ideas more readily. 

The possibility of incorporating existing systems of signs and gestures is an interesting alternative to both 
speech and text and needs further research (Dreyfuss, 1972). 

6. A First Tentative Proposal 

In our move towards a theory of dynamic abstract symbolism, we explored research in semiotics in an 
attempt to choose a grammar. Several ahematives such as the grammar of Conceptual Dependency Theory 
(Schank, 1975; Schank, & Abelson, 1977) remain to be explored. However as an anchor point for our first 
proposal we quote from An Introduction to Functional Grammar (Halliday, 1985): "Our most powerful 
conception of reality is that it consists of 'goings-on': of doing, happening, feeling, being." Halliday identifies 
the three most common types of processes: material processes, mental processes and relational processes. These 
categories seem amenable to representation. We shall begin by symbolising examp'^s of the three processes, 
considering each process in turn. 

1. Material processes: processes of doing. Here we have the common "Actor, Process and Goal" construct 
that suggests animation. We can animate actions for walk, run, jump, dance, break, make, give, take, etc. 

2. Mental processes: processes of sensing. Since these can be regarded as directed actions we shall indicate 
them by dynamic lines or arrows. For example, love can be a golden shaft directed towards the object. 
Liking will be represented by a less intense yellow while hate will be brown or black. Directed fear and 
worry will be jagged shafts. 

3. Relational processes: processes of being. Here we shall consider two classes: those that are attributes of 
the object and those which can be considered as states of the object. Attributes we shall assign as 
associated symbols as described in Section 7. States we can indicate by colouring objects, for example, 
pink for happy and blue for sad. 

7. Using Abstract Symbols 

The basic symbols we introduce will need to be simple - yet be beautiful in their own right. The more 
frequently a symbol is used the simpler it should be. Additional characteristics will be added by association and 
only when needed. Obviously we shall not have to introduce new symbols for all words: we shall include 
accepted signs, (Liungman, 1991), such as an X for "forbidden", and we can have dictionaries of known shapes 
and symbols to call on. As noted in Section 3 we shall try to avoid using shapes that are too lifelike, partly 
because this limits the reader's imagination and partly because using abstract symbols gives us more freedom to 
add attributes without contravening our built-in sense of appropriateness. For example, while it may be quite 
inappropriate to paint a life-like symbol of a beautiful woman a golden yellow colour, a golden circle definitely 
transmits the idea of beauty. 

There are two distinct facets to abstraction: abstraction of appearance and abstraction of senses (feeling, 
thinking and emotion). As described above, a golden circle may represent beauty of appearance. But beauty 
applies just as significantly to the abstract, and it is qualities such as feelings, thoughts and emotions that have 
proved difficult for existing visual forms of communication. Abstract symbols can take on additional attributes 
with no conflict. Beautiful feelings, beautiful thoughts, beautiful emotions: all can be abstractly displayed as 
golden states. Even speech (or thought) balloons which look clumsy on lifelike characters may be artistically 
attached to symbols. 

We shall begin by introducing basic symbols using the following classification system: 
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Aclor 



y \ 

Plant Animal 



O 



Inanimaio 
S/ 

/\ 
/ \ 

Mineral Building 

en 



7.1 The Addition of Attributes 

.1.= commc, b,o,„sic., .y„te,s: + ,., end („.,.,. we have s,n,bo,,c ,op,eseZ„; L 



9,.. ™, 



0 and 



(Male animal). 



Using ,he symbol - ,o rcprcscn, "i„,elligc„," (fo, "b,ai„ waves") ,hese can be f„„he, modified ,o represem- 

t (Woman) and (Man). 

7.2 Applying the Principle of Orthogonality 

The term "onhogonality" was introduced by Aad van WimpaarHen fn ^\.. a - 
languages. ,o mean features ,ha, carry across from one contex. o ^ htr^M^^lrdt To eTsfr"'"' 

we shall build into MUSLI as much orthogonality as possible. ^Wijngaarden, 1965). To ease learning 

For example, we can define: Female plant Wale plant BiseTual plant 

If we now introduce the additional attributes- i" fwedicai^ X Cw , ^ .hh fl ^ 

symbols ^ to represent a four wheeled vehicle t™ 1 a llf 

following definitions: represent a mountain, we can now form the 



Doctor 



(3 

Hospital 



Ambulance 



Medicinal plant Factory 



fa □ 6 X 

Factory School Holvbuildinn Wniw 



Holy building Holy mountain 

73 Condensing a Group of Symbols 

If .he attributes in a group are relatively constant then the original symbol plus attributes can be replaced 
by a condensed short-form We shall a frAmP f 1 »^ . • • • ^^piacea 

aE. a wo*„, (man.,:;?;;,^ ^nTeSi^r^^o SsT^^Ldrwro^^^^^^^^ 
hemic) condensing lo is rcpresenled by: ' ' O-'- » 




\ 



and » j / 



^^'"P"""'^ symbols can be condensed at any stage either by the reader n.n.ino.nH I- 
the system converting them Usuallv thr rr.ri,T Lii 1 T ■ ^^^J,' P^"''"fi ^nd clicking on them or by 

achieved in a variety of ways rc h.ns . r h ^'^"'"^ " ^^"^^"1 ^"'^ 'h''^ can be 

ancty ol ua>s. Perhaps the compound symbol will have i,s border change colour before the 
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change or both symbols may be displayed at once with the compound one gradually decreasing in size while the 
condensed symbol increases. Interesting possibilities of letting an electronic personal assistant control the 
degree and timing of the condensations remain to be explored. r • s 

The condensed symbols will be animated^ust as groups of primitives can be. A workmg (manufacturing) 

woman carrying water may be either 1^ or and might be shown moving towards a growing 

(manufacturing) plant vb. We would then, of course, see the attribute for growing increase in size! 

Again we emphasise that we are not producing cartoons: the symbols, like those shown above, will have to 
be learnt. They are not intended to be immediately obvious, but to carry meanings that will not be forgotten 
once learnt. 

7.4 Expanding Condensed Symbols 

Expanding symbols is the reverse process to condensing symbols. As mentioned in Section 3, readers will 
be able to freeze the display at any point in time and then, for example, by double-clicking on a compound 
symbol dMa^lhe^riginal primitives. Thus double-clicking on the symbol for a vehicle, may expand it 
back to: lSI!^ - where » is the symbol for motion (since a vehicle is an inanimate object moving 
people). 

8. Bringing it Ail Alive With Dynamic Sjmbols 

At this point the reader may be forgiven for wondering whether we are not simply reinventing a system of 
hieroglyphics. The difference is that people such as the Egyptians did not have ways to animate their written 
sviTibols' 

In a MUSLI document objects can move and change with time, and hence we frequently refer to it as a 
movie. Describing dynamic processes is not easy and of course we look forward to the day when we shall be 
writing a MUSLI movie instead of static text! 

We shall begin by very briefly indicating how we might dynamically represent each of the processes 
introduced in Section 6. First of all it is easy to see how we can animate the material processes of doing. 
Symbols can be made to move on the screen in ways that suggest walking, running, jumping, dancing, breaking, 
and making - for "making" we can show the parts coming together to form a whole. For the cases of giving and 
taking we suggest introducing a dynamic shaft with an arc at the end of it. On object is "given" by letting the 
shaft extend and "push" the object away. An object is "taken" when it is drawn in closer to the subject as the 

shaft shortens. . . , 

Secondly mental processes of sensing, which we represent by dynamic lines, will continuously change in 
position length, width, colour and intensity. The golden shaft of love, reaching out towards its object may 
become'more or less intense in colour as the movie continues. A child's initial fear of a doctor will be palpably 
obvious when shown by a pulsating black jagged arrow and with time the fear may fade (literally) and morph 
into the symbol for respect. 

Thirdly there are the relational processes of being. Besides the attributes that are assigneu as associated 
symbols as described in Section 7, there are the emotions, which we can indicate by shading the object dilferent 
colours Here again the dynamic nature of our medium allows emotions to change in a continuous spectrum - as 
of course they do in real life. Conflicting emotions can be beautifully depicted by showing an animated 
interplay between the colours representing rivalling emotions - both colours and intensities can interact. 

Attributes can also be dynamic. A symbol that depicts an animal slowly ascending a mountain can 

certainly suggest a very high motmtain. a • . • i 

Going one step further, symbols themselves may be formed from animated sequences. A simple circle 
representing an animal "flying" off a base line can reprcs;ent_abird flying. 



Of course this can be condensed: 
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Arid again, the condensed symbol can be animated - in this case by letting the wings move Aiid so on" 

c„„,h I f ""'''"^ '^'^'^ 'y"^"' representing "immaiure" may morph into .he 

symbol for sophisticated" as the character in a story gains experience. 

At this point we would like to change our point of view to the "other side of the mountain" and make a 
onci reference to the language for cosmic intercommunication: LINCOS (Freudenthal, 1%0). The author 
hypothesises that the "reader" can deduce meaning from the given graded series of examples. New words are 

e der wm h"h '"'^ ""'"'''^^ °^ ^^^'"P'" '^'^ '^^80 enough then the 

reader will deduce from its context the intended meaning for each newly introduced "word" (in this case a word 
IS a pattern of radio pulses). This principle can obviously be applied to MUSLI movies, particularly those that 
contam symbols for abstract ideas. Although LINCOS deliberately avoids "showing" as a means for' illustrating 
meaning, MUSLI can use dynamic symbols to represent abstract concepts such as number. We can successively 
display two plants, two animals, two vehicles, and so on, before condensing the dynamic symbol down to a 
s.atic one Once the numeric symbols have been introduced we can then dynamically introduce the symbol for 
the set of natural numbers, etc. The LINCOS exercises introducing the rules of number would in themselves 
make appropriate MUSLI movies for introducing algebra. 

VVe can also use dynamic .symbols for other abstract ideas such as "human". In this case the dynamic 
symbol would show various subsets of the set [man, woman and child] before condensing to the symbol for 
human. An idea like "medical" can be formed by demonstrating its adjectives 

We arc very aware of how much we have left undone but we look forward to delving deeper We refer the 
ulrJJr,.! n as Dynamic Graphic Interfaces: Interpreting the Meaning of a Visual 

Representation", (Jonassen, Goldmann-Segall, & Maurer, 1993), for an exposition on moving icons. 

9. Examples 

we shl ^oZe'lI'ra fiw' "'^^ " " '^^'^^ ^^--^^ P^--^^' 

Our first abstract document, produced in Macromind Director, was an exceedingly simple yet surprisindv 
rZr^ niovie of child's visit to hospital. The animated symbols show clearly the bonding betwee^ 
the mother and child. The trip into the hospital and the child's recovery were also easily animated 

v.r.-n" °7 '^7MCT?'°^"''•' ' P'^" '° P^°^"" ' presentation, and possibly a CD 

IZTl , ' ^ H . """T ^^'"^ '''P'"'" ' P''^'^ interactions between citizens and 

hoS H .?h' J ^" '° dynamic representations. At the beginning the interactions are overtly 

hostile but they change dramatically with time. The solution to the crisis is a novel one as it neither bani 
personalised transport nor enforces public systems^. 

MtlSn'mT'" ^""^ '° ^yP'=™'^^'^ 'y''"^ implemented where the user can be provided with interactive 
MUSLI movies corresponding to alternative search paths through the network - complete with symbols 
md.cating the type of links that are used and what kind of data is available on the way 

And imagine how much more enjoyable it would be coping with e-mail presented in a MUSLI setting with 
dynamic symbols - particularly if we had an electronic personal assistant to help us! ^ 

10. Conclusion 



Even a superficial survey of signs and symbols shows the richness of our semiotic heritage. The system 
w propose will be flexible enough to blend traditional forms of documentation into modern mult 
mformation re trieval systems while gradually becoming enriched with its own symbolic language MUSLI 

ijmosTvl'.T;''"''"''' '"'^'1'^' r""' '""^ '"P^ be coupled together, as requi.cd. with ,hc 

o ucimics suth as dining car.s and boutiques and may even find i.me and opportunity for an overdue haircut! 
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enables "reading" at different levels using an extended version of a reference system, and makes maximal use of 
the principle of orthogonality to aid learning. We discuss the necessary general requirements for a symbolic 
language before making a first concrete proposal based on the principles of functional grammar. We are aware 
that the mastery of any new language involves effort, but surely we can take heart from the success of other 
interactive activities such as video games. 

The system we have described is a first step towards mastering the art of electronic communication 
involving all our senses, so that we can reap the real benefits associated with a new age of hypermedia. 

It seems appropriate to conclude a paper such as this with a quote from a Gary Larson cartoon. The 
medieval soldiers are fighting to grim death on the battlements of the castle. Arrows and spears are flying thick 
as rain. Then one soldier turns to another and remarks that it doesn't really matter whether they win or lose: it's 
fun! This is how we feel about MUSLI: it is not yet clear whether the eventual outcome will be a revolution in 
communication and language, or a modest addition to the theory of semiotics. One thing is clear: MUSLI is fun. 

We hope that this first glimpse will encourage many readers to join us in further exploration of MUlti 
Sensory Language Interfaces. 
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